Content validation of a questionnaire to assess the knowledge of pediatricians, family, and community physicians on obesity

ABSTRACT Objective: The aim of this study was to validate the content of a questionnaire in order to assess the attitudes and practices in childcare consultations, knowledge on overweight and obesity, their risk factors, and barriers in addressing the issue by pediatricians and family physicians. Methods: The Delphi technique was used, with the objective of reaching a consensus on a certain subject, through experts’ opinions. The content validity index (CVI) of each item, axis, and questionnaire was calculated. The inter-rater reliability was calculated using an agreement coefficient suitable for the answer distribution such as Gwet's AC2 with ordinal weight. Results: A total of 63 experts were invited to assess and give their opinion on the questionnaire. In all, 52 accepted the invitation and analyzed the instrument. After two rounds, the questionnaire reached the proper CVI for the study and was considered complete, with its final version having 40 questions, a final index of 95%, and an inter-rate reliability of 0.905. Conclusions: This instrument, developed to assess attitudes and practices, knowledge, and barriers found in addressing the obesity by primary care physicians, obtained a CVI greater than 0.8 and an excellent agreement coefficient of the 52 judges. Therefore, its content can be considered validated.


INTRODUCTION
Obesity has become a global public health problem and was classified as the epidemic of the 21st century. In approximately 40 years, the global prevalence of overweight and obesity has increased eightfold in the 5-19 age group. 1 This has worsened in the past 20 years, with the global prevalence of children with excessive weight having jumped from 4.9 to 5.6% in 2016, reaching 340 million children aged between 5 and 19 years and 38 million children aged under 5 years in 2019, according to the World Health Organization (WHO). 2 In Brazil, the 2019 National Health Survey showed that the percentage of obese adults increased from 12.2 to 26.8% (from 2003 to 2019), while 26.1% of adolescents aged 15-19 years were overweight. 3 In the United States, obesity is associated with an annual cost increase of 36% with health and 77% with medication, compared to the expenses of people with normal weight. 4 In Brazil, in 2018, 16% of the total hospital admissions through the Unified Health System were due to comorbidities associated with obesity, such as arterial hypertension and diabetes, at a cost of BRL 3.84 billion. 5 According to a 2016 WHO survey, obesity is the fifth leading cause of death, accounting for 23% of ischemic diseases and 44% of diabetes cases globally. 6 Studies have shown that obese children and adolescents are more likely to remain obese throughout their lives. 7 Certain periods of childhood are considered critical for the development of obesity, such as the neonatal period, the first year of life, the ages between 3 and 7 years, and adolescence. 8 Evidence for the period known as "1000 days," which covers from the day of conception to the second year of life, is increasing. This phase is a window of opportunity for environmental and nutritional interventions in children at risk of obesity, which, if successful, could result in improving their body composition in the long term.
Family physicians and pediatricians are primary care professionals who, by monitoring children and their families regularly from prenatal care to adulthood, can identify overweight and obesity, thus being able to carry out an early intervention that may result in positive changes in the child's weight gain. 9 The question is whether these professionals find it difficult to detect overweight and obesity and their risk factors at an early stage. An American study applied a questionnaire to primary care physicians and found out that only 26% of them diagnosed obesity correctly, 56% thought they had received sufficient training to prevent and treat obesity, and 63% believe that they have limited time to talk about nutrition. 10 In 2006, another American study applied a questionnaire to pediatricians with the objective of evaluating attitudes, practices, and barriers found by pediatricians related to obesity. Once identified, a series of policies were proposed to address it. In 2017, the same questionnaire was applied, and an improvement was observed in most of the flaws pointed out in 2006. 11 The hypothesis raised is whether a questionnaire would be able to measure the knowledge of pediatricians and family physicians regarding the diagnosis of overweight, obesity, and their risk factors, in addition to identifying the professionals' difficulties in facing the issue. As there is no validated instrument that evaluates the practices, knowledge, and barriers related to the obesity of primary sector physicians, in Portuguese, it was decided to develop an instrument to validate its content and thus identify the aspects that, in primary health care attention, need to be improved in terms of approach, diagnosis, and prevention of overweight, with the objective of contributing to the decrease in obesity prevalence in children and adolescents in Brazil.

METHOD
This is a study with the subsequent application of questionnaires, aiming to validate an analysis instrument. The Delphi technique was used to judge the information and reach a consensus among experts on the given subject. This technique consists of a series of rounds of questionnaires with controlled feedback, allowing experts (judges) to express their opinions on a given topic, in order to find the most reliable consensus on the topic. It is a useful technique when a group's judgment is more reliable than individual opinions and has the advantage of including geographically distant participants in a simple, quick, and low-cost manner. 12 To validate the content, some questionnaires were submitted to a group of judges considered experts on the topic in question. The following inclusion criteria were applied: pediatric endocrinologist or pediatric nutrologist, as they are professionals used to deal with obesity. Professionals who do not have qualifications as specialized physicians in the area were excluded from the sample. The sample included professionals from different regions of Brazil with expertise in the area and availability to participate in the study. A convenience sample was used. We met experts known to the authors and asked for indication of other professionals. Those with the highest academic titles and used to treating patients with obesity were prioritized. A minimum number of 20 was initially stipulated to start the study. Due to the fear of dropouts during the project, 63 judges were invited.
An e-mail invitation was sent to each judge with explanations about the research objectives and the Informed Consent Form. The instrument was applied through the online tool Google Forms, and the experts had 10 days to reply with their answers.
Rev Paul Pediatr. 2023;41:e2022063 A Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree) was used to classify the experts' opinions. 12 Each question, each topic axis, and the questionnaire as a whole were evaluated as follows.
Each topic was evaluated using four questions created by the authors in order to assess the representativeness, relevance, clarity, and coherence of the content, as these items are the most commonly used criteria in the content validity assessment sheet. 12 • Is the statement clear? Can it be understood at a first reading?
• Is the answer consistent with the question? • Are the alternatives clear, without the possibility of double interpretation? • Is the question relevant, that is, is it important to know the content addressed in the question?
Each axis was evaluated through two close-ended questions and an open-ended one: • Is this axis relevant? Is it important within the study subject? • Is this axis comprehensive? Does it cover the main aspects of the topic? • Open-ended question: Are there any questions that could be included or modified within this axis? Which one?
The questionnaire as a whole was evaluated through two close-ended questions and an open-ended one: • Is this questionnaire relevant to assess the knowledge of pediatricians and family doctors about obesity diagnosis, risk factors, and the adoption of preventive measures? • Is it comprehensive? Does the questionnaire cover the main aspects to assess the knowledge about the study topic? • Open-ended question: Is there anything you would like to add to the questionnaire?
The initial questionnaire had 35 questions distributed in three axes: Axis 1: "Attitudes and practices of pediatricians or family physicians during the childcare"; Axis 2: "Theoretical knowledge about obesity diagnosis and identification of its risk factors"; and Axis 3: "Barriers found by family physicians and pediatricians to address the obesity issue during the appointments." Scientific articles, guidelines from the Brazilian Society of Pediatrics, and the WHO were used as references for the preparation of the questions. [13][14][15][16][17][18][19][20][21] At the end of each round, the content validation index (CVI) was measured. To finish the Delphi technique rounds and validate the questionnaire, the CVI should be greater than 80%.
The CVI measures the percentage of experts who agree on certain aspects of the instrument. The score is calculated by adding up items that were marked with a "4" or "5" by the experts, dividing this number by the total number of responses, and multiplying the result by 100. New Delphi rounds were carried out until reaching 80% CVI.
In the characterization of the experts, the categorical variables were described by absolute and relative frequencies and the numerical variables by means or medians and standard deviations or interquartile range.
In the evaluation phase using the Delphi technique, the experts' answers were classified by absolute frequencies and percentages, to verify the clarity of alternatives and statements, the coherence between statements and alternatives, and the relevance and pertinence of the questions.
The questionnaire evaluations were described by the percentages of each type of answer. The CVI was calculated per item (CVI-i), per axis (CVI-a), and for the full questionnaire (CVI-q), according to the aspect evaluated. The CVI-a is the average of the CVI-i of each item of the questions referring to the analyzed axis, and the CVI-q is the average of the CVI-i of each item evaluated in all the questions. The analysis was performed using the SPSS software, version 24. Data were collected and stored in Google Drive and exported to Microsoft Excel, where the analyses were performed.
The agreement among the experts was calculated using an inter-rater reliability measurement appropriate to the answer distribution, such as Gwet's AC2 with ordinal weight, 22 with the coefficients followed by 95% confidence intervals and p-values for the test of hypotheses. The agreement coefficients were compared to the classification presented by Altman, 23 which considers poor the coefficients lower than 0.2, reasonable between 0.2 and 0.4, moderate between 0.4 and 0.6, good between 0.6 and 0.8, and excellent those above 0.8. For this study, CVI values or agreement coefficients over 0.80 were considered satisfactory. The analyses were performed using the irrCAC, an R package. 24 The survey project was submitted for analysis by the Survey Ethics Committee and approved by Statement No. 4

RESULTS
A total of 63 experts were invited to assess and give their opinion on the questionnaire. In all, 52 accepted the invitation and analyzed the instrument, and 11 accepted the invitation but were unable to respond within the deadline. At the beginning of the questionnaire, the experts were asked the following question: "Do you think obesity is handled well by family physicians or pediatricians?" 65.4% disagreed. The data of the 52 experts are presented in Table 1.
In the first round, the experts expressed their opinions for each item of the 35 questions, according to the Likert scale. Questions 8 and 9 did not reach the minimum CVI of 80% for clarity of alternatives, and question 31 did not reach the CVI for clarity of statement. Also, 100% of the experts found the three axes relevant, and 98% found them comprehensive. Regarding the questionnaire, 98% rated it as relevant and comprehensive.
After the statistical analysis and reading experts' comments, the questionnaire was reformulated. In addition to the questions with CVI<80%, questions 1, 29, and 30 were modified based on the experts' opinions, even having reached CVI>80%. The Likert scale of axis 1 was changed based on suggestions, and four new questions were added. The changes were submitted for analysis by the experts in a second round.
In the second round, the same 52 experts analyzed and answered the questionnaire. In total, 86% of them agreed to modify the Likert scale of axis 1. All modified questions achieved a CVI greater than 80%, and the four added questions obtained a CVI above 90%.
After two rounds, the questionnaire reached the CVI appropriate for the study, with its final version containing 40 questions. The final CVI of all questions was over 80%. The CVI of the axis and the final questionnaire can be found in Table 2.
After the two rounds, the expert agreement index was calculated using Gwet's AC2 inter-rater reliability coefficient. In the second round, the agreement indexes for clarity of the statements, clarity of the alternatives, coherence, relevance, and general agreement were excellent. After the final changes, the inter-rater reliability was calculated again with 40 questions, and the data are shown in Table 3.

DISCUSSION
The minimum requirement for the development of a measurement instrument is to have its content validated as relevant and representative. 25 The validation process is one of the most common challenges in the preparation of this type of instrument. This study faced this challenge and validated the content of a questionnaire that will serve as an instrument to assess the attitudes, the practices, and the knowledge of family physicians and pediatricians in relation to the diagnosis, prevention, and treatment of obesity, as well as to identify the barriers found by these professionals to address this topic. The validity of a measurement tool is a critical factor in its selection and application in both professional practice and academic research. An instrument is valid when it measures what it was supposed to measure. 26 The content is validated by a panel of experts who judge the instrument elements and categorize the tool according to the relevance and representativeness of its content. 27 There is no consensus in the literature on the minimum number of experts. Grant and Davis do not specify an exact number, but often the studies use a group of up to 10 experts. These authors also propose that the decision on the number of experts depends on their level of specialization, their experience, and their knowledge on the subject being evaluated. 28 This study invited 63 experts. There was also a concern about the quality of the experts. Half of them had a doctorate or a postdoctoral degree, setting up a panel of experts with a high degree of specialization. Additionally, they came from all regions of Brazil, contributing opinions based on their different regional realities.
At the end of the study, more than 80% of the experts responded not only to the first round but also to the second. This high number of experts could have been a problem for the study, as it is known that the higher the number of people evaluating a topic, the harder it is to reach a consensus, which could lead to many evaluation rounds. Nevertheless, this was not the case, and a consensus was reached after just two rounds.
The low abstention observed in this study can be explained by the use of an online form that facilitated the experts' answers and reminded them about the evaluation each week. Another reason that may justify such adherence is the fact that the prevalence of obesity has been increasing in recent years and that its negative impact on individual and collective health has encouraged the experts to contribute with an instrument that can improve the management of such disease in the health system. 1,10,11 Upon the receipt of the experts' answers, the questionnaires were subjected to quantitative and qualitative analyses. There is more than one approach to verify the validity of an instrument content, as well as several statistical methods for data analysis. Some studies classify these methods into two categories: indexes related to content validity and indexes of general agreement. 29 This study used both methods for the analysis.
Polit and Beck proposed a CVI>0.78 for each item of the questionnaire and a mean CVI>0.9 for the content to be considered validated. 30 This study required an index >0.8 for both measures. At the end of the study, the CVI of all instrument items was above 0.8. The mean CVI of all axes and the mean CVI of the final questionnaire were both above 0.9, which allowed the content of the instrument to be considered validated and representative.
There are several ways to calculate the agreement coefficient. This study chose to use Gwet's AC2, which is more appropriate when there are concentrations of responses in one direction, that is, when the proportion of favorable and unfavorable responses is not the same. 22 After all the modifications, the experts agreed that the statements and alternatives were clear and that the questionnaire was coherent and relevant, as all these items had an index greater than 0.8 with statistical significance. The final agreement coefficient was excellent, reaching 0.905 in the general analysis, indicating that, despite the high number of experts, they agreed with each other.
As limitations of the study, we can mention the general scope of the questionnaire, which was 0.78 in the first round. A possible justification for this index lower than 0.8 may be the fact that not all the causes of obesity were addressed in the questionnaire -only the main ones, because obesity is multifactorial, and it was not possible to include all causes. 13 Conducting a pilot test with the application of the final instrument to the target audience could also bring improvements to the tool. The lack of face-to-face meetings with the experts to clarify doubts regarding the evaluation steps could negatively contribute to the results. Therefore, to minimize this risk, the guidelines of some authors were followed for the instrument validation process, such as sending an invitation letter, the references, the study objective, and a detailed instruction manual on how to proceed with the evaluation. The validation process is more comprehensive than content validation. 26 Therefore, it is necessary to subject the questionnaire to other psychometric tests, such as reliability or construct validation, for example, before applying it in research.
The development of a questionnaire that can assess whether family physicians and pediatricians know how to diagnose obesity and overweight, whether they know how to identify their risk factors, and what are the barriers they find in addressing the issue can be useful in improving pediatric patient care and adopting effective preventive measures against obesity, as these professionals are inserted in a strategic position in the public health system, which is the primary sector. 9 These physicians are essential in the detection of obesity, the application of preventive measures, and the assistance in and implementation of public policies to fight the increased prevalence of obesity.
This developed instrument had a CVI>0.8, and an excellent agreement coefficient from the judges, which validate the instrument. Therefore, the instrument can be considered relevant to identify aspects that need to be improved with regard to the approach, diagnosis, and prevention of overweight in primary health care attention.